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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  DEFENSE  (COMMAND, 

CONTROL,  COMMUNICATIONS,  AND 
INTELLIGENCE) 

DIRECTOR,  DEFENSE  SECURITY  ASSISTANCE  AGENCY 

SUBJECT:  Audit  Report  on  the  Defense  Security  Assistance  Management  System 
(Report  No.  98-095) 


We  are  providing  this  audit  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  rqwrt  in  preparing  the  final  rQwrt. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
Therefore,  we  request  that  the  Defense  Security  Assistance  Agency  provide  additional 
comments  on  Recommendations  2.a.,  2.d.,  2.e.,  2.f.,  2.g.,  2.h.,  and  2.i.  in  response 
to  the  final  report  by  May  26,  1998. 


We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 
should  be  directed  to  Ms.  Evelyn  R.  Klemstine,  Audit  Program  Director,  at 
(703)  604-9172  (DSN  664-9172)  (eklemstine@dodig.osd.mil).  See  Appendix  D  for  the 
r^iort  distribution.  The  audit  team  members  are  listed  inside  the  back  cover. 


David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 
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Defense  Security  Assistance  Management  System 
Executive  Summary 

Introduction.  The  Defense  Security  Assistance  Management  System  (DSAMS)  is  a 
worldwide  automated  information  system  that  will  be  used  for  managing  the  annual  $10  billion 
Foreign  Military  Sales  Program.  As  envisioned,  it  will  replace  13  automated  information 
systems  used  by  the  Military  Departments  and  Defense  Security  Assistance  Agency  and  will 
provide  standardized  and  improved  security  assistance  management  that  will  reduce  cost  and 
eliminate  redundant  procedures.  The  Defense  Security  Assistance  Agency  is  responsible  for 
developing  and  operating  DSAMS  and  maintaining  the  worldwide  database  required  by  all 
levels  of  management  for  the  security  assistance  program.  In  February  1995,  the  Defense 
Security  Assistance  Agency  awarded  a  contract  to  BDM  Enterprise  to  prepare  a  conceptual 
design  document  for  DSAMS.  The  conceptual  design  document  estimated  life-cycle  costs  for 
DSAMS  to  be  $58.3  million  and  estimated  completion  of  the  system  by  May  1999.  In 
August  1995,  the  Director,  Defense  Security  Assistance  Agency,  approved  the  development  of 
DSAMS,  based  on  the  conceptual  design  document.  A  Program  Management  Office  within 
the  Defense  Security  Assistance  Agency  is  responsible  for  the  acquisition,  development, 
implementation,  and  maintenance  of  DSAMS. 

Audit  Objectives.  The  overall  audit  objective  was  to  determine  whether  DSAMS  will  meet 
user  requirements  and  cost,  schedule,  and  performance  parameters.  We  also  reviewed 
management  controls  as  they  related  to  the  audit  objective. 

Audit  Results.  The  DSAMS  was  not  being  managed  with  controls  appropriate  to  a  system  of  its 
cost  and  size.  Based  on  information  provided  by  Defense  Security  Assistance  Agency,  the 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence)  did  not 
classify  DSAMS  as  a  major  automated  information  system  and  there  has  been  no  Major 
Automated  Information  System  Review  Council  oversight  to  ensure  that  essential  management 
control  documents  were  prepared.  The  Defense  Security  Assistance  Agency  had  not  prepared  a 
mission  needs  statement,  an  operational  requirements  document,  a  program  baseline,  an 
acquisition  strategy,  an  acquisition  plan,  and  a  test  and  evaluation  master  plan  to  help  manage  the 
cost,  schedule,  and  performance  parameters  of  the  system.  As  a  result,  the  Defense  Security 
Assistance  Agency  fielded  four  versions  of  the  case  development  module  that  were  not  fully 
operational,  resulting  in  program  slippages  fi'om  May  1999  to  beyond  the  Year  2000.  In  addition, 
there  are  no  assurances  that  future  DSAMS  modules  will  meet  user  requirements  md  cort, 
schedule,  and  performance  parameters.  Further,  program  slippages  have  resulted  in  requiring  an 
addition^  $2.8  million  to  make  existing  systems  Year  2000  compliant. 


Summary  of  Rcconimendatioiis.  We  recommend  that  the  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and  Intelligence)  (ASD[C^I])  classify  the  DSAMS  as  a 
major  automated  information  system  and  provide  program  oversight.  We  recommend  that  the 
Defense  Security  Assistance  Agency  submit  the  life-cycle  cost  estimate  to  the  ASp(C^I)  for 
reclassification  of  DSAMS  as  a  major  automated  information  system  and  to  the  Assistant 
Secretary  of  Defense  (Program  Analysis  and  Evaluation)  for  validation  of  the  accuracy  and 
completeness  of  the  life-<^cle  cost  estimates;  update  the  life-cycle  cost  estimates  for  each 
milestone  review;  prepare  a  mission  needs  statement,  an  operational  requirements  document,  a 
program  baseline,  an  acquisition  strategy,  an  acquisition  plan,  and  a  test  and  evaluation  master 
plan;  and  establish  an  integrated  product  team. 

Management  Comments.  The  Office  of  the  ASD(C^I)  did  not  concur  vwth  the  recommendation 
on  classifying  DSAMS  at  this  time.  It  stated  that  it  will  reconsider  its  position  after  re\dewing  the 
Institute  for  Defense  Analyses  independent  cost  estimate  due  in  April  1998.  In  the  interim,  it  will 
place  DSAMS  on  its  list  of  information  technology  initiatives  subject  to  review  by  the  DoD  Chief 
Information  Officer.  The  Acting  Director,  Defense  Security  Assistance  Agency,  nonconcurred 
with  submitting  the  Institute  for  Defense  Analyses  final  report  to  the  ASD(C*I)  for  reclassification 
of  DSAMS  because  DSAMS  development  costs  were  below  the  dollar  threshold  required  for  a 
Major  Automated  Information  System  Review  Council  review.  The  Acting  Director  concurred 
with  all  other  reconunendations.  The  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Financial 
Management  and  Comptroller)  also  commented  and  stated  that  the  mission  needs  should  also 
include  requirements  to  serve  the  financial  management  and  accounting  communities.  A 
discussion  of  management  comments  is  in  Part  I  and  the  complete  text  is  in  Part  III. 

Audit  Response.  The  ASD(C^I)  comments  were  responsive.  DoD  Regulation  5000.2-R 
requires  DoD  Components  to  rqwrt  the  initial  estimated  life-cost  and  notify  the  ASD(C  I) 
when  cost  growth  or  a  change  in  acquisition  strategy  results  in  reclassifying  an  acquisition 
program.  Thus,  the  DSAA  should  submit  the  life-cycle  cost  estimate  for  DSAMS  to  the 
ASD(C^I).  Although  the  Acting  Director  concurred,  we  consider  the  comments  partially 
responsive  and  additional  comments  are  required  on  coordinating  the  required  documentation 
approvals  through  the  proper  channels,  and  including  representatives  from  the  Offices  of  the 
ASD(C^I)  and  the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation)  and 
selected  members  of  the  financial  management  community  on  the  integrated  product  team.  We 
request  that  the  Defense  Security  Assistance  Agency  reconsider  its  position  and  provide 
additional  comments  in  response  to  the  final  report  by  May  26,  1998. 


ii 


Table  of  Contents 


Executive  Summary 


Fart  I  -  Audit  Results 

Audit  Background 
Audit  Objectives 

Management  of  the  Defense  Security  Assistance  Management  System 

Fart  n  -  Additional  Information 

Appendix  A.  Audit  Process 
Scope  and  Methodology 
Management  Control  Program 
Summary  of  Prior  Coverage 
Appendix  B.  Existing  Security  Assistance  Systems 
Appendix  C.  Modules  Within  the  Defense  Security  Assistance 
Management  System 
Appendix  D.  Rqwrt  Distribution 

Fart  ni  -  Management  Comments 

Office  of  the  Assistant  Secreta^  of  Defense  (Command,  Control, 
Communications,  and  Intelligwice)  Comments 
Defense  Security  Assistance  Agency  Comments 
Assistant  Secretary  of  the  Air  Force  Comments 


2 

3 

4 


20 

21 

21 

23 

24 

25 


28 

29 

38 


Part  I  -  Audit  Results 


Audit  Background 

The  Defense  Security  Assistance  Management  System.  The  Defense  Security 
Assistance  Management  System  (DSAMS)  will  become  the  automated  information 
system  used  worldwide  for  managing  the  annual  $10  billion  Foreign  Military  Sales 
Program.  As  envisioned,  DSAMS  will  use  state-of-the-art  technology  and  replace 
13  Military  Department  and  Defense  Security  Assistance  Agency  (DSAA)  security 
assistance  automated  information  systems  (see  Appendix  B).  The  development  of  ^ 
DSAMS  involves  the  integration  of  selected  features  from  each  of  the  existing  systems 
and  commercial  off-the-shelf  software.  Once  implemented,  DSAA  expects  DSAMS  to 
reduce  the  annual  automated  data  processing  operation  and  maintenance  cost  of 
$36.5  million  by  $23  million  for  manipulating,  processing,  and  producing  foreign 
military  sales  information.  In  addition,  DSAMS  will  eliminate  redundant  procedures 
that  the  Military  Departments  are  using  for  processing  and  reporting  that  information. 
The  Defense  Information  Systems  Network  will  be  the  communication  link  between  the 
server  located  at  the  Defense  Information  Systems  Agency  megacenter  in  Oklahoma 
City  and  more  than  50  user  sites  in  the  United  States.  DSAA  is  responsible  for 
developing  and  operating  DSAMS  and  maintaining  the  worldwide  database  required  by 
all  levels  of  management  for  the  security  assistance  program.  In  August  1995,  the 
Director,  DSAA,  approved  the  development  of  DSAMS  using  Foreign  Military  Sales 
administrative  surcharge  funds^  to  finance  DSAMS.  A  program  management  office, 
established  in  September  1995,  within  DSAA  is  responsible  for  the  acquisition, 
development,  implementation,  and  maintenance  of  DSAMS. 

Acquisition  of  DSAMS.  The  DSAMS  will  consist  of  five  modules:  case 
development,  case  implementation,  case  execution,  case  reconciliation  and  closure,  and 
training  (see  Appendix  C).  Using  an  incremental^  development  approach,  the 
acquisition  of  DSAMS  has  been  segregated  into  five  stages:  feasibility,  analysis, 
design,  construction,  and  implementation.  In  February  1995,  the  DSAA  awarded  a 
contract  to  BDM  Biterprise  to  prepare  a  conceptual  design  document  for  DSAMS.  The 
conceptual  design  document  estimated  life-cycle  costs  for  DSAMS  to  be  $58.3  million 


^For  the  purpose  of  this  report,  existing  systems  are  the  13  Military  Department  and 
DSAA  automated  information  systems  that  DSAMS  will  replace. 

^e  administrative  surcharge  fund  is  a  collection  of  the  3  percent  additive  to  the  cost 
of  Defense  articles  and  services  charged  to  the  foreign  customer  to  recoup  the  costs 
associated  with  the  administration  of  the  sale. 

^In  an  incremental  acquisition  approach,  part  of  the  system  is  fielded  while  the 
remaining  parts  are  being  developed  in  a  parallel  or  subsequoit  effort. 
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and  estimated  completion  by  May  1999.  The  development  of  the  case  development 
module  began  in  S^tember  1995.  The  Military  Departments  and  DSAA  are  testing  the 
case  development  module. 

Acquisition  Management.  The  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  (ASD[(^I]),  based  on  input  provided  by 
DSAA,  classified  DSAMS  as  a  nonmajor  automated  information  system.  As  a 
nonmajor  automated  information  system,  the  Director,  DSAA,  was  designated  as  the 
Milestone  Decision  Authority.  The  Milestone  Decision  Authority  is  the  executive 
authority  over  developmental,  operational,  and  standardization  improvement  efforts. 

The  DSAMS  Program  Manager  directs  the  central  design  activity  (including  the 
contractor),  chairs  the  Configuration  Control  Board,  and  executes  configuration 
management  control  of  DSAMS  and  the  existing  systems.  The  Program  Manager  also: 

o  estimates  the  life-cycle  cost  of  DSAMS  and  provides  cost  information  to  the 
Milestone  Decision  Authority; 

o  identifies  user  requirements  to  determine  cost,  schedule,  and  performance 
parameters;  and 

o  assesses  program  risks,  optimizing  system  performance  and  minimizing  the 
cost  of  ownership. 


Audit  Objectives 

The  overall  audit  objective  was  to  determine  whether  DSAMS  will  meet  user 
requirements  and  cost,  schedule,  and  performance  parameters.  We  also  reviewed 
management  controls  as  they  related  to  the  audit  objective.  See  Appendix  A  for  a 
discussion  of  the  audit  scope  and  methodology,  our  review  of  the  management  control 
program,  and  a  summary  of  prior  coverage  related  to  the  audit  objective. 
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Management  of  the  Defense  Security 
Assistance  Management  System 

The  DSAMS  is  not  being  managed  with  controls  appropriate  to  a  system  of  its 
cost  and  size.  Based  on  information  provided  by  DSAA,  the  ASD(C  I)  did  not 
classify  DSAMS  as  a  major  automated  information  system  (MAIS)  and  there 
has  been  no  MAIS  Review  Council  (MAISRC)  oversight  to  ensure  that  essential 
management  control  documents  were  prepared.  DSAA  had  not  prepared  a 
mission  needs  statement,  an  operational  requirements  document,  a  program 
baseline,  an  acquisition  strategy,  an  acquisition  plan,  and  a  test  and  evaluation 
master  plan  to  help  manage  the  cost,  schedule,  and  performance  parameters  of 
the  system.  DSAMS  was  not  classified  as  a  MAIS  because  the  Program 
Management  Office  did  not  adequately  estimate  and  report  life-cycle  cost  when 
ASD(C^I)  was  categorizing  the  system,  causing  the  system  to  be  misclassified  as 
a  nonmajor  automated  information  system  and  exempting  it  from  MAISRC 
review.  In  addition,  the  Program  Management  Office  did  not  have  controls  in 
place  to  establish  an  integrated  product  team  and  to  define  and  manage  user 
requirements.  As  a  result,  DSAA  fielded  four  versions  of  the  case  development 
module  that  were  not  fiilly  operational,  resulting  in  program  slippages  from 
May  1999  to  beyond  the  Year  2000.  In  addition,  there  were  no  assurances  that 
future  DSAMS  modules  will  meet  user  requirements  and  cost,  schedule,  and 
performance  parameters.  Further,  program  slippages  have  resulted  in  requiring 
an  additional  $2.8  million  to  make  existing  systems  Year  2000  compliant. 


Guidance 

Defense  Federal  Acquisition  Regulation  Supplement,  Part  207.  The  Defense 
Federal  Acquisition  Regulation  Supplement,  part  207,  “Acquisition  Planning,”  requires 
that  an  acquisition  plan  be  prepared  when  total  costs  of  all  contracts  for  a  program  are 
estimated  at  $15  million  annually  or  at  $30  million  throughout  the  life  of  the  program. 

DoD  Directive  5000.1.  DoD  Directive  5000.1,  “Defense  Acquisitions,” 

March  15,  1996,  provides  broad  policies  and  principles  for  all  DoD  acquisition 
programs,  and  establishes  a  disciplined,  yet  flexible,  management  approach  for 
acquiring  and  managing  automat^  information  systems.  The  Directive  establishes 
responsibilities  for  DoD  Component  heads  that  ensure  policies  and  procedures 
governing  the  operation  of  the  Component’s  acquisition,  budgeting,  and  requirements 
systems  are  effectively  implemented.  The  Directive  summarizes  the  primary  objective 
of  a  Defense  acquisition  as: 

...  to  acquire  qiulity  products  that  satisfy  the  needs  Of  the  operational  user 
with  measurable  inyrovements  to  mission  accomplishment,  in  a  timely 
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manner,  at  a  &ir  and  reasonable  price.  Successful  acquisition  programs  are 
fundamentally  dq>endent  upon  conq>etent  people,  rational  priorities,  and 
clearly  defined  responsibilities.  The  following  policies  and  principles  govern 
the  operation  of  the  defense  acquisition  system  and  are  divided  into  three  major 
categories:  (1)  Translating  Operation  Needs  into  Stable,  Affordable  Programs, 

(2)  Acquiring  Quality  Products,  and  (3)  Organizing  for  Efficiency  and 
Effectiveness.  These  principles  shall  guide  all  defense  acquisition  programs. 

DoD  Regulation  5000.2-R.  DoD  Regulation  5000.2-R,  “Mandatory  Procedures  for 
Major  Defense  Acquisition  Program  and  Major  Automated  Information  System 
Acquisition  Programs,"  March  15,  1996,  establishes  mandatory  procedures  for  MAIS 
acquisition  programs  regardless  of  funding  source.  It  defines  a  MAIS  as  an  automated 
information  system  that  is  estimated  to  require  program  costs  in  any  single  year  in 
excess  of  $30  million,  total  program  costs  in  excess  of  $120  million,  or  total  life-cycle 
costs  in  excess  of  $3^  million.  Those  automated  information  systems  not  meeting  the 
thresholds  are  classified  as  nonmajor  automated  information  system  programs. 

Automated  Information  System  Oversight.  A  MAIS  acquisition  program  is 
subject  to  oversight  by  the  MAISRC,  which  is  chaired  by  ASD(C^).  The  Awistant 
Secretary  also  serves  as  the  Chief  Information  Officer  and  the  Milestone  Decision 
Authority  for  a  MAIS.  The  Chief  Information  Officer  is  responsible  for  monitoring  the 
performance  of  information  technology  programs,  evaluating  the  performance  of  those 
programs,  and  advising  the  head  of  the  agency  whether  to  continue,  modify,  or 
terminate  a  program.  For  a  nonmajor  automated  information  system  acquisition,  the 
component  head  rather  than  the  MAISRC  is  responsible  for  management  oversight 
during  the  development,  operational,  and  standardization  improvement  efforts. 

Automated  Information  System  Requirements.  DoD  Regulation  5000. 2-R 
describes  broad  management  principles  that  are  applicable  to  major  and  nonmajor 
acquisition  programs.  It  provides  procedures  and  required  documentation  for 
translating  broadly  stated  mission  needs  into  a  well-defined,  carefully  structured 
program  to  reflect  a  balance  between  cost,  schedule,  and  performance.  In  addition,  the 
operational  performance  objectives  and  minimum  acceptable  requirements  are  to  be 
documented  to  include  a  mission  needs  statement,  an  operational  requirements 
document,  a  program  baseline,  an  acquisition  strategy,  and  a  test  and  evaluation  master 
plan.  It  requires  management  to  structure  the  automated  information  system  acquisition 
to  ensure  a  logical  progression  through  a  series  of  phases  designed  to  reduce  risk, 
ensure  affordability,  and  provide  adequate  information  for  decisionmaking  that  will 
provide  the  need  in  the  shortest  practical  time.  To  ensure  that  deployed  systems  satisfy 
user  requirements,  the  Secretary  of  Defense  had  directed  the  use  of  integrated  product 
teams  to  facilitate  the  oversight  and  review  of  the  life-cycle  management 
documentation. 

DoD  Manual  5000.4-M.  DoD  Manual  5000.4-M,  “Cost  Analysis  Guidance  and 
Procedures,”  December  1992,  establishes  guidance  for  preparing  cost  analysis 
documents  to  be  used  as  a  basis  for  preparing  life-cycle  cost  estimates.  The  Manual 
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defines  life-cycle  cost  as  all  cost  categories  from  conceptual  design  to  the  end  of  the 
system’s  useful  life  regardless  of  fund  source  or  management  control.  The  Manual 
requires  that  the  program  office  estimate  and  prq>are  a  cost  analysis  in  support  of 
acquisition  milestone  reviews. 


Designation  as  a  Msyor  Automated  Information  System 

The  DSAMS  is  not  being  managed  with  appropriate  controls.  The  ASD(C^I)  did  not 
classify  DSAMS  as  a  MAIS.  In  August  1995,  DSAA  estimated  DSAMS  life-cycle  cost 
of  $58.3  miUion.  In  October  1996,  ASD(C^I)  included  DSAMS  on  its  MAIS 
acquisition  program  list.  However,  in  November  1996,  DSAA  informed  ASD(C  I) 
that  significant  factors  prevented  the  inclusion  of  DSAMS  as  a  MAIS.  Specifically, 
DSAA  stated: 

The  DSAMS  is  not  a  DoD  operational  system,  having  no  impact  on  DoD 
domestic  missions  and  functions.  Funding  for  the  DSAMS  is  non- 
appropriated,  coming  entirely  from  the  Foreign  Military  Sales  (FMS) 
Administrative  Trust  Fund.  The  estimated  funding  requiremrat  for  the 
development  of  DSAMS  is  substantially  below  the  current  MAISRC  threshold 
of  $120miUion,  as  are  the  life  cycle  costs  below  the  current  MAISRC 
threshold  of  $360  million.  Finally,  the  administrative  requirements  of 
documenting  a  MAISRC  process  would  significantly  impact  the  limited 
manpower  and  resources  we  have  currently  committed  to  this  project. 

In  its  memorandum  to  ASD(C^I),  DSAA  stated  that  nonappropriated  funding  precluded 
classifying  DSAMS  as  a  MAIS  acquisition.  However,  DoD  Regulation  5000.2-R  does 
not  exempt  an  acquisition  program  based  on  the  funding  source.  In  addition,  DSAA 
stated  that  DSAMS  costs  were  substantially  below  the  threshold  of  a  MAIS.  However, 
when  DSAA  prepared  the  cost  estimate  for  DSAMS,  the  estimated  costs  were  solely  for 
software  development  of  the  case  development  module.  Based  on  the  model  used  by 
the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation),  the  estimated 
life-cycle  costs  for  DSAMS  is  about  $500  million.  DSAMS  ultimately  exceeded  the 
life-cycle  cost  threshold  for  a  MAIS. 


Estimating  and  Reporting  Life-Cycle  Cost 

The  DSAMS  was  not  classified  as  a  MAIS  because  the  Program  Management  Office 
did  not  adequately  estimate  and  report  life-cycle  cost  when  the  system  was  being 
categorized  by  ASD(C^I),  causing  the  system  to  be  misclassified  as  a  nonmajor,  and 
exempting  it  from  MAISRC  review.  In  August  1995,  the  Program  Management  Office 
estimated  life-cycle  cost  of  $58.3  million  for  DSAMS.  In  October  1997;  however,  the 
Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation)  model  estimated 
life-cycle  costs  of  about  $500  million.  Life-cycle  costs  include  the  cost  for  acquisition. 
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implementation,  and  operation  of  the  system.  Identifying  and  developing  adequate 
life-cycle  cost  is  critical  to  many  facets  of  program  management,  including  ev^uating 
the  system’s  cost-effectiveness  and  determining  the  amount  and  timing  of  funding 
requirements.  In  addition,  life-cycle  costs  are  used  as  a  basis  for  approval  by  the 
Milestone  Decision  Authority  before  proceeding  to  the  next  acquisition  phase. 
Estimating  and  reporting  accurate  costs  are  necessary  for  successful  system  control  and 
development  as  weU  as  for  identifying  the  Milestone  Decision  Authority  level  for 
approval. 

DSAA  Estimate  of  Life-Cycle  Costs.  The  Program  Management  Office  estimated 
Ufe-cycle  cost  of  $58.3  million  for  DSAMS  based  on  BDM  Enterprise’s  cost  for 
developing  the  case  development  module  software.  It  did  not  include  all  critical  costs 
of  the  system.  Specifically,  the  estimate  did  not  include  the  acquisition  costs  of  the 
other  four  modules.  In  addition,  it  did  not  include  program  management  personnel, 
operation  and  maintenance,  construction  for  central  design  activity,  and  DSAMS 
interface  with  other  systems. 

Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation)  ^imate  of 
Life-Cycle  Costs.  An  Assistant  Secretary  of  Dpfense  (Program  Analysis  ^d 
Evaluation)  model  estimated  life-cycle  costs  of  $500  million,  based  on  estimated  source 
lines  of  code^  totaling  2.5  million  provided  by  DSAA.  DoD  Directive  5000.1  i^uires 
the  Office  of  the  Assistant  Secretary  of  Defense  (Program  An^ysis  and  Evaluation)  to 
determine  the  accuracy  and  completeness  of  life-cycle  cost  estimates  for  automated 
information  systems.  The  Office  of  the  Assistant  Secretary  of  Defense  (Program 
Analysis  and  Evaluation)  uses  the  System  Evaluation  and  Estimation  of  Resources  - 
Software  Estimation  model  to  validate  DoD  Component  life-cycle  cost  estimates  and  to 
evaluate  vendor  proposals  for  DoD  automated  information  system  contracts.  The 
model  considers  industry  averages,  previous  DoD  system  acquisitions,  and  total  source 
lines  of  code  to  estimate  software  development  and  maintenance  cost.  The  most 
significant  advantage  of  using  source  lines  of  code  to  estimate  total  life-cycle  costs  is 
that  those  codes  are  directly  related  to  the  software  to  be  built.  The  model  estimated 
life-cycle  cost  of  $500  million,  thereby  exceeding  the  $360  million  life-cycle  cost 
threshold  for  a  MAIS. 


Program  Documentation 

There  has  been  no  review  at  the  Office  of  the  Secretary  of  Defense  level  to  ensure  that 
essential  management  control  documents  were  prepared.  Specifically,  DSAA  did  not 
prepare  a  mission  needs  statement,  an  operational  requirements  document,  a  program 

^Source  lines  of  code  are  statements  that  provide  control,  logic,  Md  mathematical 
instructions  to  the  computer.  Each  statement  is  counted  to  obtain  the  total  source  lines 

of  code. 
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baseline,  an  acquisition  strategy,  an  acquisition  plan,  and  a  test  and  evaluation  master 
plan  to  help  manage  the  cost,  schedule,  and  performance  parameters  of  the  system. 
Although  DSAMS  was  not  classified  as  a  MAIS,  DoD  Directive  5000. 1  and 
DoD  Regulation  5000.2-R  require  similar  program  documentation  for  the  acquisition  of 
major  and  nonmajor  automated  information  systems.  To  properly  manage  the 
acquisition  of  any  automated  information  system,  documentation  is  necessary  to  ensure 
a  logical  progression  through  a  series  of  phases  designed  to  reduce  risk,  ensure 
affordability,  and  provide  adequate  information  for  decisionmaking.  As  of 
November  14,  1997,  the  Program  Management  Office  had  not  prepared  the  r^uired 
documentation,  even  though  system  development  was  in  milestone  II  phase  2  of  the 
acquisition  process. 


Defining  and  Managing  User  Requirements 

The  DSAA  did  not  prepare  the  necessary  program  documentation  for  DSAMS  because 
controls  were  not  in  place  to  define  and  manage  user  requirements.  Controls  are 
comprehensive  strategies  that  provide  reasonable  assurance  that  programs  and 
administrative  and  operating  ftinctions  are  efficiently  and  effectively  carried  out  in 
accordance  with  applicable  law  and  management  policy.  User  requirements  of  an 
automated  information  system  program  affect  cost,  schedule,  and  performance  of  the 
system  development.  Sufficiently  defined  and  managed  requirements  are  essential  to 
the  success  of  an  automated  information  system  acquisition. 

Defining  Requirements.  Controls  for  defining  user  requirements,  which  determine  the 
minimum  operational  capability  of  the  system,  were  not  in  place.  Defining  user 
requirements  is  the  process  of  translating  broadly  stated  mission  needs  into  a  set  of 
operational  requirements  from  which  cost,  schedule,  and  performance  parameters  are 
derived.  DoD  Regulation  5000.2-R  describes  how  those  parameters  should  be 
presented  and  documented  in  a  mission  needs  statement  and  an  operational 
requirements  document.  A  mission  needs  statement  identifies  the  needs  of  users, 
describes  the  deficiencies  in  the  existing  systems,  and  identifies  potential  alternatives. 
An  operational  requirements  document  contains  system  cost,  schedule,  and 
performance  parameters  based  on  the  requirements  established  in  the  mission  needs 
statement. 

Mission  Needs  Statement  and  Operational  Requirements  Document.  The 
Program  Management  Office  had  not  prepared  a  mission  needs  statement  and  an 
operational  requirements  document,  specifying  DSAMS  operational  requirements. 
Without  a  mission  needs  statement  and  an  operational  requirements  document,  the 


^Milestone  n  phase  2  of  the  acquisition  process  is  the  development  phase  in  which  the 
overall  software  design  is  coded  and  tested  to  ensure  user  requirements  are  satisfied. 
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Program  Manager  could  not  determine  whether  operational  performance  requirements 
were  effectively  satisfied  by  all  users  of  the  system  and  whether  cost,  schedule,  and 
performance  parameters  for  DSAMS  were  met  or  exceeded.  During  the  development 
of  DSAMS,  the  Military  Department  requirements  were  not  standardized  for  the  case 
development  module,  which  resulted  in  the  development  of  additional  versions  of  the 
module  to  satisfy  user  needs.  In  addition,  DSAA  and  officials  from  the  Defense 
Finance  and  Accounting  Service  did  not  adequately  coordinate  DSAMS  financial 
reporting  requirements  relating  to  billing  the  foreign  military  sales  customers  for 
delivered  Defense  articles  and  services. 

Conceptual  Design  Document.  As  an  alternative  to  the  mission  needs 
statement  and  operational  requirements  document,  the  Program  Management  Office  had 
BDM  Enterprise  prqwre  a  concq)tual  design  document.  However,  the  document  did 
not  fully  satisfy  the  requirements  of  DoD  Regulation  5000. 2-R  because  it  did  not 
present  the  users  needs,  all  significant  deficiencies  in  the  existing  systems,  and 
alternatives  to  develop  a  new  system.  In  addition,  the  conceptual  design  document  did 
not  present  the  cost,  schedule,  and  performance  parameters  for  DSAMS  as  a  total 
system  and  presented  only  the  parameters  for  the  case  development  module. 

Managing  Requirements.  Controls  for  managing  user  requirements  were  not  in  place 
to  establish  an  integrated  product  team  and  to  effectively  assess  the  impact  of 
requirements  on  system  cost,  schedule,  and  performance  parameters.  DoD 
Regulation  5000.2-R  requires  the  use  of  integrated  product  teams  to  perform  oversight 
and  review  of  the  automated  information  system  acquisition  and  preparation  of  a 
program  baseline,  an  acquisition  strategy,  and  a  test  and  evaluation  master  plan.  The 
Defense  Federal  Acquisition  Regulation  Supplement  requires  that  an  acquisition  plan  be 
prepared  and  used  as  a  tool  for  managing  and  controlling  the  acquisition  process. 

Integrated  Product  Teams.  The  DSAA  had  not  established  an  integrated 
product  team  to  provide  oversight  and  review  as  DSAMS  proceeds  through  its 
acquisition  life-cycle.  Integrated  product  teams  are  an  integral  part  of  the  Defense 
acquisition  oversight  and  review  process.  An  integrated  product  team  helps  build 
successful  and  balanced  programs,  identify  and  resolve  issues,  and  make  sound  and 
timely  recommendations  to  freilitate  decisionmaking.  The  goal  of  the  team  is  to 
resolve  as  many  issues  and  concerns  at  the  lowest  level  possible,  and  to  expeditiously 
escalate  issues  that  need  resolution  at  a  higher  level,  bringing  only  the  highest  level 
issues  to  the  Milestone  Decision  Authority  for  decision.  The  integrated  product  teams 
consist  of  qualified  team  members  from  the  appropriate  functional  disciplines,  who  are 
empowered  to  make  commitments  for  the  organizations  or  functional  areas  that  they 
r^resent. 

The  DSAMS  integrated  product  team  should  have  comprised,  at  a  minimum, 
representatives  from  ASDfC^I),  the  Defense  Finance  and  Accounting  Service,  the 
Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation),  and  the  Military 
Departments.  The  ASD(C^I)  is  responsible  for  overseeing  all  DoD  information 
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technology  investments,  including  MAIS  acquisition  programs.  The  Defense  Fmance 
and  Accounting  Service  is  responsible  for  accounting,  bUling,  and  financial 
management  rqwrting  for  the  security  assistance  program.  The  Assistant  Secretary  of 
Defense  (Program  Analysis  and  Evaluation)  provides  guidance  as  well  as  reviews  the 
results  of  the  analysis  of  alternative  studies  for  acquisition  programs,  and  determines 
whether  the  cost  and  benefit  analyses  are  accurate  and  complete.  The  Military 
Departments  execute  the  security  assistance  programs. 

Program  Baseline.  The  Program  Management  Office  did  not  establish  a 
program  baseline  for  DSAMS  to  identify  cost,  schedule,  and  performance  parameters. 
DoD  Regulation  5000. 2-R  requires  the  program  office  to  develop  a  program  baseline  at 
program  initiation,  at  each  subsequent  major  milestone,  and  following  a  program 
restructuring  or  deviation.  The  program  baseline  is  used  as  a  management  technique  to 
manage  and  control  cost  growth  and  schedule  slippages.  Although  the  conceptual 
design  document  presented  the  cost,  schedule,  and  performance  parameters  for  the  case 
development  module,  it  did  not  present  those  parameters  for  DSAMS  as  a  total  system. 
Without  a  program  baseline,  the  Program  Manager  could  not  identify  and  report 
significant  deviations  from  planned  cost,  schedule,  and  performance  parameters. 

Acquisition  Strategy.  The  Program  Management  Office  did  not  develop  an 
acquisition  strategy  to  minimize  the  time  and  cost  to  satisfy  user  requirements,  identify 
the  system’s  life  cycle  costs  and  resource  requirements,  and  manage  risk  for  the  system 
development.  The  acquisition  strategy  was  not  prepared  for  the  total  system.  An 
acquisition  strategy  is  the  overall  plan  that  a  Program  Manager  follows  throughout  the 
acquisition  process.  It  also  serves  as  the  contract  between  tee  Program  Manager  and 
subordinate  or  supporting  organizations  and  specifies  what  is  needed  and  expected  from 
those  organizations  during  tee  acquisition  process.  Without  an  acquisition  strategy ,  tee 
Program  Manager  cannot  control  all  elements  of  the  DSAMS  acquisition  process, ^ 
including  tee  plans  to  verify  teat  user  requirements  were  satisfied.  As  an  alternative  to 
tee  acquisition  strategy,  DSAA  had  BDM  Enterprise  prepare  a  software  development 
plan  for  tee  case  development  module;  however,  tee  plan  did  not  include  tee  remaining 
four  modules  of  tee  system.  In  addition,  tee  software  development  plan  did  not 
identify  tee  system’s  life  cycle  and  what  resources  will  be  used. 

Acquisition  Plan.  The  Program  Management  Office  had  not  develop^  an 
acquisition  plan  to  identify  program  risks  and  solutions  to  those  risks.  At  a  minimum, 
tee  contract  amount  of  $58.3  million  for  tee  case  development  module  exceeded  the 
$30  milUon  criteria  established  in  tee  Defense  Federal  Acquisition  Regulation 
Supplement  for  preparing  an  acquisition  plan.  The  Progr^  Management  Office 
considered  the  conceptual  design  document  as  tee  acquisition  plan;  however,  teat 
document  did  not  identify  tee  risk  associated  with  cost,  schedule,  and  performance 
parameters  as  well  as  solutions  to  those  risks  so  teat  program  goals  could  be  met. 
According  to  DSAA  officials,  conversion  of  tee  data  fi-om  tee  existing  systems  to 
DSAMS  was  far  more  complex  than  originally  thought.  Without  an  acquisition  plan, 
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the  Program  Management  Office  could  not  adequately  identify  and  assess  program 
requirements  to  reduce  schedule  slippages  and  additional  costs  to  the  program. 

Test  and  Evaluation  Master  Plan.  The  Program  Management  Office  had  not 
developed  a  test  and  evaluation  master  plan  for  DSAMS  to  establish  the  criteria  for 
testing  the  system  to  verify  that  user  requirements  are  met.  As  an  alternative  to  the  test 
and  evaluation  master  plan,  DSAA  had  BDM  Enterprise  pr^>are  the  software  test 
description  and  software  test  plan  for  the  case  development  module.  However,  the 
alternative  documents  addressed  the  actual  detailed  test  and  evaluation  plan  for  the  case 
development  module  and  did  not  address  the  overall  test  and  evaluation  for  DSAMS. 
During  the  Air  Force  testing  of  the  case  development  module  software,  over 
300  deficiencies  were  identified,  including  mistitled  field  labels;  extremely  slow  system 
q)eed;  and  overlapping  text  on  document  printouts. 


Cost,  Schedule,  and  Performance  Risk 

Versions  of  the  Case  Development  Module.  The  DSAA  fielded  four  versions  of  the 
case  development  module  that  were  not  fully  operational,  resulting  in  program 
slippages  from  May  1999  to  beyond  the  Year  2000.  From  November  1996  through 
October  1997,  the  Program  Management  Office  was  required  to  make  changes  to  the 
functionality  of  the  case  development  module  because  the  module  did  not  meet  user 
requirements.  Four  versions  of  the  module  have  been  developed  to  meet  user 
requirements.  The  first  version  provided  line  item  totals  and  rounded  dollar  values  up. 
The  second  version  validated  the  source  of  supply  and  provided  military  articles  and 
services  list  descriptions.  The  third  version  provided  amendment  and  modification 
calculations.  The  fourth  version  included  a  field  for  national  stock  numbers.  The 
implementation  of  the  case  development  module  slipped  more  than  a  year;  and 
DSAMS,  as  a  total  system,  has  slipped  more  than  2  years  with  no  agreement  as  to  the 
actual  fielding  of  DSAMS.  The  following  figure  shows  slippages  in  the  estimated 
completion  date  of  DSAMS  as  stated  in  various  source  documents. 
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1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006 


Estimated  Fielding  Date 

^The  Conceptual  Design  Document  (CDD)  approved  by  the  milestone  decision  authority 
in  August  1995. 

^Minutes  of  the  December  1995  Executive  Steering  Committee  (ESC)  meeting. 
^Minutes  of  the  March  1 996  ESC  meeting. 

^he  Information  Technology  Management  Strategic  Plan  (ITMSP)  dated  June  1997. 


DSAMS  Estimated  Completion  Dates 


As  illustrated  in  the  figure  above,  estimates  for  the  completion  of  DSAMS  were 
inconsistent.  Without  controls  in  place  to  define  and  manage  user  requirements,  there 
was  no  effective  mechanism  to  monitor  DSAMS  cost,  schedule,  and  performance 
parameters;  thus,  there  were  no  assurances  that  future  DSAMS  modules  would  meet 
user  requirements  and  cost,  schedule,  and  performance  parameters. 

Year  2000  Compliance.  Because  the  fielding  of  DSAMS  has  slipped,  DSAMS  will 
not  be  fully  operational  by  the  Year  2000  and  existing  systems  will  need  to  become 
Year  2000  compliant.  In  the  Year  2000,  automated  information  systems  that  are  not 
Year  2000  compliant  will  have  to  be  adjusted  for  the  way  dates  are  recorded  and 
computed.  For  the  past  several  decades,  systems  typically  used  two  digits  to  represent 
the  year,  such  as  “97”  rq)resenting  1997,  in  order  to  conserve  on  electronic  data 
storage  and  to  reduce  operating  costs.  With  the  two-digit  format,  however,  the 
Year  2000  is  indistinguishable  fiom  1900.  The  calculation  of  dates  is  further 
complicated  because  the  Year  2000  is  a  leap  year,  being  divisible  by  both  100  and  400, 
while  the  year  1900  is  not.  As  a  result  of  the  ambiguity,  system  and  application 
programs  that  use  dates  to  calculate,  compare,  or  sort  could  generate  incorrect  results 
when  working  with  years  after  1999. 

In  April  1997,  ASD(C:’l)  issued  the  “DoD  Year  2000  Management  Plan.”  According 
to  the  management  plan,  DoD  Components  are  responsible  for  renovating  existing 
systems  to  fix  the  Year  2000  deficiency  by  December  1998.  Although,  a  memorandum 
of  agreement  was  not  prepared  between  DSAA  and  the  Military  Departments,  DSAA 


12 


Management  of  the  Defense  Security  Assistance  Management  System 


agre^  in  October  1996  to  provide  funds  to  fix  the  existing  systems  Year  2000 
deficiencies  because  of  delays  in  fielding  DSAMS.  DSAA  estimates  that  $2.8  million 
will  be  required  to  make  existing  systems  Year  2000  compliant.  DSAA  has  provided 
the  Military  Dq)artments  the  necessary  funds  and  the  Military  Departments  are  in  the 
process  of  fixing  the  Year  2000  deficiencies. 


Management  Initiative 

The  Program  Management  Office  has  taken  action  to  obtain  an  adequate  life-cycle  cost 
estimate  for  DSAMS.  In  August  1997,  the  Program  Management  Office  awarded  a 
contract  to  the  Institute  for  Defense  Analysis  to  perform  a  complete  cost  analysis, 
including  a  life-cycle  cost  estimate  of  the  total  DSAMS.  The  analysis  will  be  available 
in  April  1998. 


Conclusion 

Effective  controls  over  the  acquisition  of  DSAMS  are  critical  to  the  program’s  success. 
The  long-term  success  of  DSAMS  is  contingent  on  identifying  life-cycle  costs,  defining 
overall  mission  needs  and  requirements;  developing  an  acquisition  strategy  to  verify 
that  user  requirements  are  met;  identifying  measurable  means  to  monitor  and  evaluate 
costs,  schedules,  and  performance;  and  developing  a  test  and  evaluation  strategy  to 
verify  that  user  requirements  are  met.  The  life-cycle  costs  for  DSAMS  exceeded  the 
$360  million  limit  as  calculated  by  the  Assistant  Secretary  of  Defense  (Program 
Analysis  and  Evaluation),  Therefore,  DSAMS  should  be  classified  as  a  MAIS,  which 
will  result  in  increased  oversight  and  controls  on  program  management  necessary  to 
reduce  the  risks  associated  with  the  systems  development.  Until  those  issues  are  clearly 
resolved,  DSAMS  may  fail  to  meet  its  ultimate  objective  of  being  the  only  DSAA 
automated  information  system  for  managing  the  DoD  security  assistance  program. 


Management  Comments  on  the  Finding  and  Audit  Response 

Defense  Security  Assistance  Agency  Comments.  The  DSAA  disagreed  with  the 
finding  that  DSAMS  was  not  being  managed  with  appropriate  controls.  DSAA  stated 
that  the  estimated  $500  million  life-cycle  cost  for  DSAMS  was  significantly  overstated. 
The  life-cycle  cost  calculations  were  based  on  an  inaccurate  total  of  2.5  million  lines  of 
code  provided  by  DSAA.  DSAA  stated  that  a  more  accurate  estimate  of  total  lines  of 
code  could  be  determined  by  adding  together  all  the  systems  of  a  single  Military 
Department  that  cover  the  full  range  of  FMS  functionality  in  DSAMS.  Using  that 
methodology,  DSAMS  would  contain  about  1  line  of  code  for  every  1.5  lines  of  lepcy 
code  or  1  million  lines  of  codes  for  the  total  Navy  FMS  systems.  Based  on  an  Institute 
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for  Defense  Analyses  preliminary  rqwrt,  the  estimated  development  cost  of  DSAMS 
will  be  about  $100  million.  Regarding  scheduling  and  testing,  DSAA  stated  that  only 
one  version  of  the  case  development  module  was  fielded  into  production;  and  that  the 
statement,  “during  Air  Force  testing  of  the  case  development  module  software,  over 
300  deficiencies  were  identified,”  was  misleading.  It  stated  that  the  deficiencies  found 
were  either  legitimate  bugs,  which  were  subsequently  corrected,  or  changes  to  the 
design  based  on  new  user  requirements  not  previously  identified.  Finally,  DSAA 
stated  that  the  estimated  completion  date  for  DSAMS  shown  in  the  Estimated  Fielding 
Date  table  in  the  report  should  be  year  2001,  not  year  2004,  because  the  2004  date  was 
the  projected  end  of  the  system  life  cycle. 

Audit  Response.  Life-cycle  costs  are  the  total  cost  for  an  automated  information 
system  over  its  full  life.  It  includes  the  cost  of  requirements  analyses,  design, 
development,  acquisition  and  lease,  operations,  support,  and  disposal.  It  encompasses 
contract  and  in-house  costs,  all  cost  categories,  and  all  related  appropriations.  The 
$100  million  estimated  by  the  Institute  for  Defense  Analyses  is  for  developmental  costs 
and  did  not  include  other  elements  of  life-cycle  cost.  DSAA  stated  that  1  million  lines 
of  codes  was  a  more  accurate  estimate  using  the  total  lines  of  code  for  Navy  FMS 
systems  rather  than  the  2.5  million  lines  of  code  previously  provided.  Based  on  our 
review  of  the  total  lines  of  code  for  each  of  the  Military  Departments,  the  Air  Force 
FMS  systems  have  the  most  lines  of  code  while  the  Navy  systems  have  the  least  lines 
of  code.  Using  the  same  process  and  ratios  DSAA  used  in  calculating  the  Navy  FMS 
systems,  the  Air  Force  systems,  adjusted  for  Defense  Finance  and  Accounting  Service 
functionality  and  inflated  by  lOOK  to  cover  unknowns,  is  approximately  2  million  lines 
of  code.  Consequently,  the  $100  million  cost  data  are  incomplete,  thus  the 
$500  million  estimated  life-cycle  cost  for  DSAMS  may  not  be  significantly  oversmted. 
However,  until  a  complete  life-cycle  cost  estimate  is  validated,  we  cannot  determine 
whether  DSAMS  meets  the  MAISRC  dollar  threshold  requirements. 

We  disagree  with  the  DSAA  assessment  that  deficiencies  in  the  four  versions  of  the 
case  development  module  were  based  on  new  requirements.  The  enhancements  and 
deficiencies  found  were  critical  functions  in  processing  FMS  to  accomplish  the  same 
efficiency  as  the  legacy  system  that  was  omitted  during  the  planning  process.  As  stated 
in  the  rq>ort,  the  four  versions  were  developed  to  meet  user  requirements.  Some  of 
those  deficiencies  may  have  been  identified  had  a  mission  needs  statement  and 
operational  requirements  document  been  prepared  and  approved  prior  to  system 
development. 

During  the  audit  field  work,  the  DSAMS  Program  Management  Office  was  unable  to 
provide  us  an  estimated  completion  date  for  DSAMS.  In  addition,  DSAA  stated  that  it 
could  not  estimate  an  overall  completion  date  until  after  the  implementation  of  the  case 
development  module.  As  an  alternative,  the  December  1995  minutes  for  the  fifth 
Executive  Steering  Committee  meeting  showed  an  estimated  completion  date  for 
DSAMS  of  August  31,  1999,  while  the  March  1996  minutes  for  the  sixth  Executive 
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Steering  Committee  meeting  showed  an  estimated  completion  date  of 
November  30,  2004.  The  Executive  Steering  Committee  meeting  minutes  do  not 
support  the  year  2004  as  the  projected  end  of  the  system  life  cycle.  In  addition,  the 
conceptual  design  document  for  DSAMS  showed  a  useful  life  of  10  years,  not  3  years. 


Recommendations,  Management  Comments,  and  Audit  Response 

1.  We  recommend  that  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  classify  the  Defense  Security  Assistance 
Management  System  (DSAMS)  as  a  nutior  automated  information  system  and 
provide  program  oversight. 

ASD(C^  Comments.  The  Office  of  the  ASD(C^I)  did  not  concur  at  this  time  with  the 
recommendation  and  stated  that  it  will  reconsider  its  position  after  a  review  of  the 
Institute  for  Defense  Analyses  independent  cost  estimate,  which  is  due  in  April  1998. 

In  the  interim,  the  Office  of  the  ASD(C’l)  will  place  DSAMS  on  its  list  of  information 
technology  initiatives  subject  to  review  by  the  DoD  Chief  Information  Officer. 

Audit  Response.  We  consider  the  proposed  actions  responsive. 

2.  We  recommend  that  the  Director,  Defense  Security  Assistance  Agency: 

a.  Submit  the  life-cycle  cost  estimate  prepared  by  the  Institute  for  Defense 
Analysis  to  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  for  reclassification  of  DSAMS  as  a  major 
automated  information  system. 

b.  Submit  the  life-cycle  cost  estimate  prepared  by  the  Institute  for  Defense 
Analysis  to  the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 
for  validation  of  the  accuracy  and  completeness  of  the  life-cycle  cost  estimates. 

c.  Update  the  life-cycle  cost  estimate  for  each  milestone  review  so  that 
DSAMS  can  be  evaluated  for  its  cost-effectiveness. 

d.  Prepare  a  mission  needs  statement  for  DSAMS  to  identify  users  needs, 
describe  the  deficiencies  in  the  existing  systems,  and  identify  potential  alternatives. 

e.  Prepare  an  operational  requirements  document  for  DSAMS  to  establish 

system  cost,  schedule,  and  performance  parameters  based  on  the  user 
requirements. 

f.  FcfrfthKdi  an  integrated  product  team  to  provide  oversight  and  review  as 
DSAMS  proceeds  through  its  acquisition  life  cycle.  The  integrated  product  team 
should  include,  at  a  minimum^  representatives  from  the  Office  of  the  Assistant 
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Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence),  the 
Defense  Finance  and  Accounting  Service,  the  Office  of  the  Assistant  Secretary  of 
Defense  (Program  Analysis  and  Evaluation),  and  the  Mihtary  Departments* 

g.  'ii’jrfaMigh  a  program  baseline  for  DSAMS  to  manage  and  control  cost 
grovrth  and  schedule  slippages. 

h.  Prepare  an  overall  acquisition  strategy  for  DSAMS  to  manage  user 
requirements  and  identify  the  system’s  life  cycle  and  the  resources  to  be  used. 

i.  Prepare  an  acquisition  plan  for  DSAMS  to  identify  the  risk  associated 
with  cost,  schedule,  and  performance  parameters  as  well  as  solutions  to  those  risks 
so  that  program  goals  can  be  met. 

j.  Prepare  a  test  and  evaluation  master  plan  for  DSAMS  to  establish  the 
criteria  for  testing  the  system  to  verify  that  user  requirements  are  met. 

DSAA  Comments.  The  Acting  Director,  DSAA,  nonconcurred  with 
Recommendation  2.a.,  stating  that  DSAMS  costs  are  under  the  MAISRC  threshold;  and 
that  in  its  November  19,  1996,  memorandum,  the  ASD(C®I)  agreed  that  DSAMS  was 
below  the  MAISRC  threshold.  However,  DSAA  will  submit  the  Institute  for  Defense 
Analyses  final  report  to  ASD(C^I)  for  reclassification  of  DSAMS,  should  that  cost 
estimate  meet  the  MAISRC  threshold. 

DSAA  concurred  with  all  other  recommendations  as  follows. 

-  The  Institute  for  Defense  Analyses  final  report  is  due  in  April  1998  and  costs 
will  be  tracked  and  evaluated  regularly. 

-  DSAA  has  prepared  a  mission  needs  statement,  operational  requirements 
document,  program  baseline,  acquisition  strategy,  and  acquisition  plan. 

-  The  Software  Test  Description  and  Software  Test  Plan  were  prepared  for  the 
case  development  module  following  the  standards  of  MIL-STD-498.  Further ,  the  test 

and  evaluation  master  plans  future  modules  are  case  implementation, 

September  30,  1998;  training,  November  30,  1998;  and  case  execution  and  case 
closure,  September  30,  1999. 

-  The  Executive  Steering  Committee  serves  as  the  overarching  integrated 
product  team.  That  committee  is  composed  of  representatives  from  all  appropriate 
disciplines  and  is  supported  by  a  variety  of  other  teams,  including  conceptual  design, 
general  design,  and  standardization  working  groups. 
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Audit  Response.  We  consider  the  DSAA  comments  on  Recommendation  2.a.  to  be 
nonresponsive,  because  DSAA  will  submit  the  life-cycle  cost  estimate  for  DSAMS  only 
when  the  costs  meet  the  Major  Automated  Information  System  Review  Council  dollar 
threshold  requirements.  We  acknowledge  that  ASD(C^I)  already  agreed  that  DSAMS 
was  not  a  MAISRC  system  based  on  estimated  life-cycle  cost  of  $58.3  nullion  reported 
at  that  time.  However,  the  $58.3  million  estimated  costs  were  solely  for  software 
development.  In  addition,  the  Institute  for  Defense  Analyses  preliminaiy  cost  estimate 
of  $100  million  is  also  for  development  cost  of  DSAMS.  Both  estimated  costs  are 
incomplete  because  DSAA  had  not  included  other  elements  of  life-cycle  cost.  DoD 
Regulation  5000.2-R  requires  DoD  Components  to  report  the  initial  estimated  life-cycle 
cost  and  notify  the  ASD(C^I)  when  cost  growth  or  a  change  in  acquisition  strategy 
results  in  reclassifying  an  acquisition  program.  Therefore,  we  request  that  DSAA 
submit  the  life-cycle  cost  estimate  prqwred  by  the  Institute  for  Defense  Analysis  to  the 
ASD(C^I)  for  review  as  well  as  any  change  in  cost  growth  and  acquisition  strategy. 

Although  DSAA  prepared  the  mission  needs  statement,  operational  requirements 
document,  program  baseline,  acquisition  strategy,  and  acquisition  plan,  it  did  not  meet 
the  intent  of  Recommendations  2.d.,  2.e.,  2.g.,  2.h.,  and  2.i.  because  the  documents 
haH  not  been  coordinated  and  approved  through  the  proper  channels.  Therefore,  we 
request  that  DSAA  provide  completion  dates  for  those  documents  in  response  to  the 
final  rq>ort. 

Although  DSAA  concurred  with  Recommendation  2.f.,  its  response  did  not  meet  the 
intent  of  the  recommendation.  We  agree  that  the  Executive  Steering  Committee  serves 
as  the  overarching  integrated  product  team;  however,  it  does  not  include  representatives 
from  the  Offices  of  the  ASD(C^I),  and  the  Assistant  Secretary  of  Defense  (Program 
Analysis  and  Evaluation)  and  selected  members  of  the  financial  management 
community.  The  integrated  product  team  should  include  representatives  from  all 
appropriate  functional  disciplines  working  together  to  build  a  successful  and  balanced 
program,  identify  and  resolve  issues,  and  make  sound  and  timely  recommendations  to 
facilitate  decisionmaking.  The  Air  Force  in  its  comments  also  agreed  that  it  is  critical 
that  the  integrated  product  team  include  members  from  the  fmancial  management 
community.  We  request  that  DSAA  reconsider  including  representatives  from  those 
offices  on  the  committee. 

Air  Force  Comments.  The  Air  Force  was  not  required  to  do  so,  but  responded  to  the 
draft  finding  and  recommendations.  The  Air  Force  stated  that  the  mission  needs  should 
also  include  requirements  to  serve  the  financial  management  and  accounting 
communities  and  that  financial  management  and  accounting  requirements  should  also  be 
established  as  performance  parameters.  In  addition,  it  stated  that  the  integrated  product 
team  should  include  Defense  Finance  and  Accounting  Service  and  selected  members  of 
the  frnancial  management  community  to  provide  oversight  and  review  as  DSAMS 
proceeds  through  its  acquisition  life  cycle. 
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Part  n  -  Additional  Information 


Appendix  A.  Audit  Process 


Scope  and  Methodology 

We  reviewed  the  iwogram  management  controls  in  place  to  ensure  that  DSAMS  will 
meet  user  requirements.  Specifically,  we  reviewed  DSAA  life-cycle  cost  estimates  to 
ensure  that  they  included  all  DoD  standard  cost  elements.  In  addition,  we  reviewed 
DSAMS  program  documentation  covering  the  period  from  February  1995  through 
October  1997  to  msure  compliance  with  DoD  Directive  5000. 1  and  DoD 
Regulation  5000.2-R.  Further,  we  reviewed  the  process  DSAA  used  to  manage  and 
oversee  the  acquisition  and  development  of  DSAMS.  In  addition,  we  interviewed 
individuals  and  evaluated  procedures  for  milestone  review  and  approval  for  meeting 
cost,  schedule,  and  performance  parameters.  Finally,  we  evaluated  DSAA  efforts  for 
making  the  existing  systems  Year  2000  compliant. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  nor 
statistical  sampling  to  perform  this  audit. 

Use  of  Technical  Assistance.  Our  Technical  Assessment  Division  assisted  us  in 
evaluating  technical  documentation.  In  addition,  the  Assistant  Secretary  of  Defense 
(Program  Analysis  and  Evaluation)  assisted  us  in  estimating  life-cycle  cost  for  DSAMS 
using  the  System  Evaluation  and  Estimation  of  Resources  -  Software  Estimation  model. 
Although  we  did  not  perform  a  formal  reliability  assessment  of  the  model,  our 
Technical  Assessment  Division  did  not  find  errors  that  would  preclude  the  ux  of  the 
model  estimate  to  meet  the  audit  objectives  or  that  would  change  the  conclusions  in  the 
report.  Further,  our  Technical  Assessment  Division  agreed  the  model  estimate  was  a 
conservative  estimate. 

Audit  Type,  Dates,  and  Standards.  We  performed  this  program  audit  from 
June  through  October  1997  in  accordance  with  auditing  standards  that  the  Comptroller 
General  of  the  United  States  issued,  as  implemented  by  the  Inspector  General,  DoD. 
Accordingly,  we  included  tests  of  management  controls  considered  necessary. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and  organizations 
within  the  DoD  and  BDM  Enterprise.  Further  details  are  available  upon  request. 
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Management  Control  Program  Review 

DoD  Directive  5010.38,  “Management  Control  Program,”  August  26,  1996,  requires 
DoD  organizations  to  implement  a  comprehensive  system  of  management  controls  that 
provides  reasonable  assurance  that  programs  are  operating  as  intended  and  to  evaluate 
the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  reviewed  the  adequacy  of 
DSAA  management  controls  over  DSAMS  acquisition  and  development  process. 
Specifically,  we  evaluated  the  implementation  of  DoD  policies  and  procedures 
governing  the  acquisition  and  development  of  DSAMS.  We  reviewed  management’s 
self-evaluation  applicable  to  those  management  controls. 

Adequacy  of  Management  Controls.  We  identified  material  management  control 
weaknesses  for  DSAA  as  defined  by  DoD  Directive  5010.38.  The  DSAA  management 
controls  for  DSAMS  acquisition  were  not  in  place  to  adequately  estimate  and  report 
life-cycle  cost,  to  establish  an  integrated  product  team,  and  to  define  and  manage  user 
requirements  to  ensure  that  necessary  program  documentation  was  prepared.  All 
recommendations  in  this  report,  if  implemented,  will  provide  adequate  controls  for 
defining  and  managing  user  requirements  to  develop  adequate  cost  estimates  and  to 
prepare  necessary  program  documentation.  A  copy  of  the  report  will  be  provided  to 
the  senior  official  responsible  for  management  controls  in  DSAA. 

Adequacy  of  Management’s  Self-Evaluation.  The  DSAA  officials  did  not  identify 
the  DSAMS  acquisition  and  development  as  an  assessable  unit  and,  fterefore,  did  not 
identify  or  report  the  material  management  control  weaknesses  identified  by  the  audit. 


Summary  of  Prior  Coverage 

No  prior  audit  coverage  on  the  acquisition  management  of  DSAMS  was  performed 
within  the  last  5  years.  However,  the  Inspector  General,  DoD,  issued  a  draft  report 
related  to  integrated  product  teams  discussed  in  this  report. 

Inspector  General,  DoD,  Report  No.  98-057,  “The  Defense  Fmance  and 
Accounting  Service  Acquisition  Program  for  the  Electronic  Document 
Management  Program,”  January  27, 1998,  was  requested  by  the  Director  of  the 
Defense  Finance  and  Accounting  Service  to  review  the  implementation  of  the  program 
and  input  during  the  acquisition  process.  The  Defense  Finance  and  Accounting  Service 
developed  the  required  life-cycle  documentation  and  has  positioned  Increment  1 , 

Vendor  Pay,  of  the  program  to  obtain  a  milestone  HI  deployment  decision.  The 
integrated  product  teams  identified  cost,  funding,  and  testing  concerns  that  needed  to  be 
resolved  before  a  deployment  decision  could  be  recommended.  The  Program 
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Management  Office  provided  a  cost  reconciliation  document,  funding  information,  and 
a  schedule  for  testing.  The  report  made  no  recommendation  because  management  took 
responsive  action  to  suggestions  made  during  the  review  of  milestone  III 
documentation. 
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Appendix  B.  Existing  Security  Assistance  Systems 


The  DSAMS  will  replace  the  following  13  Military  Department  and  DSAA  automated 
information  systems. 

Army: 

Army  Security  Assistance  Training  Management  System 
Centralized  Integrated  System  for  International  Logistics 
Security  Assistance,  Automation,  Army 
Security  Assistance  Case  Tracking  System 

Navy: 

Management  Information  System  for  International  Logistics 

Navy  Security  Assistance  Data  System 

Student  Training  and  Tracking  Information  System 

Air  Force: 

Case  Management  Control  System 

Security  Assistance  Management  Information  System 

Training  Control  System 

DSAA: 

Foreign  Military  Sales  Data  Base 
Foreign  Military  Sales  Credit  System 

Military  Assistance  Program/Foreign  Military  Financing  Program  and 
Delivery 

Originally,  DSAMS  was  to  replace  the  Defense  Integrated  Financial  System  of  the 
Defense  Finance  and  Accounting  Service.  However,  the  Defense  Finance  and 
Accounting  Service  did  not  agree  to  use  DSAMS  for  financial  reporting  and  was 
therefore  removed  from  the  list. 
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Module _ 

Case  development 


Case  implementation 


Case  execution 


Case  reconciliation 
and  closure 


Training 


_ DSAMS  Expected  Function - 

The  case  development  module  will  be  used  to  prepare  new 
foreign  military  sales  cases  and  to  make  modifications  and 
amendments  to  existing  cases. 

The  case  implementation  module  will  be  used  to  process 
customer  acceptance  of  foreign  military  sales  cases  and  to  issue 
case  management  directions  to  the  Military  Departments 
responsible  for  managing  the  case. 

The  case  execution  module  covers  all  of  the  processes  from  the 
preparation  of  requisitions  through  the  reporting  of  delivered 
items  or  services.  This  module  is  the  most  complex  and  largest 
module  of  DSAMS. 

The  case  reconciliation  and  closure  module  covers  the 
actions  necessary  to  close  a  case  once  it  is  logistically  and 
financially  complete.  Those  actions  include  reconciling  logistics 
and  financial  records,  resolving  discrepancies,  and  closing  the 
case. 

The  training  module  includes  all  processes  for  training  from 
customer  request  through  case  closure. 


•A  case  is  a  foreign  military  sales  contractual  agreement  between  the  United  States  and 
an  eligible  foreign  country  or  international  organization,  documented  by  a  letter  of 
offer  and  acceptance. 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
D^uty  Comptroller  (Program/Budget) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications  and  Intelligoice) 
Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 

Assistant  Secretary  of  Defense  (Public  Affairs) 

Department  of  the  Army 

Auditor  General,  Dq)artment  of  the  Army 

Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 

Auditor  General,  Dq)artment  of  the  Navy 

Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Auditor  General,  Department  of  the  Air  Force 
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Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Information  System  Agency 
Director,  Defense  Logistic  Agency 
Director,  Defense  Security  Assistance  Agency 
Director,  National  Security  Agency 
Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 

Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 
General  Accounting  Office 
National  Security  and  International  Affairs  Division 
Technical  Information  Center 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  Government  Management,  Information,  Technology, 
Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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al 


Office  of  the  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and 
Intelligence)  Comments 


OFFICE  OF  THE  ASSISTANT  SCC9ICTARY  OF  DEFENSE 
•OOO  DEFKNSK  FCNTAOOH 
WASHINGTON.  O.C.  30301 <6000 


fiB  m 


MEMORANDUM  FOR  DIRECTOR.  READINESS  AND  LOGISTICS  SUPPORT 
DIRECTORATE.  OIO 

SUBJECT:  Aoctit  Report  on  the  Defense  Security  AisUtance  Management  System 
Project  No.  72G-0034 


We  have  reviewed  the  draft  report  of  the  Inqiector  General  Readiness  and 
Logistics  Support  Duectoiile  on  the  Defense  Security  Assistance  Managenienc  System. 

We  do  not  ooocur  at  this  tiine  with  the  request  to  classify  DS  AMS  as  a  Major 
Automation  Infonnation  System.  We  will  reconsider  this  position  after  we  review  the 
results  of  lDA*s  independent  analysis,  which  it  due  to  be  published  in  April  1998.  in  the 
interim  we  will  place  DSAMS  on  our  list  of  infonnation  technology  initiatives  subject  to 
review  by  the  DoD  Chief  Infocmaiion  Officer. 

Please  contact  my  action  officer  Ms.  Angela  Bruce,  at  70a>695-090d  or 
bnicgadbQtd.penttyQn  jnil  to  discuss  the  issues. 


Anthony  M.  Valletta ' 

Deputy  Assistant  Secretary  of  Defense 
<C31  Acquisition) 


o 
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Defense  Security  Assistance  ^ency  Conunents 


DEFENSE  SECURITY  ASSISTANCE  AGENCY 

WA$HlNOTtJN.DCa030t.M00  0  9  1533 

In  fcply  refer  to: 
1-59001/97 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Audit  Report  on  the  Defense  Security  Anistsoce  Management  System 
(PiojectNo.7LG-00M) 

Reference  the  Department  of  Defense  Inspector  General's  memorandum  dated 
December  16,1 997»  regarding  the  above  subject  DSAA  has  addressed  each  of  the 
audit's  recommendations  for  oocrective  action. 

DSAA  cannot  concur  with  die  conclusion  being  drawn  by  the  draft  audit  report, 
that  the  Defense  Security  Assittance  Management  System  (DSAMS)  U  not  being 
managed  with  appropriate  controls.  The  audit’s  recommendations  fall  into  two  areas:  (I) 
that  DSAMS  will  cost  $500M  over  its  life-cycle  ($250M  development)  and  thus  should 
be  re-classified  as  a  major  automaied  information  system;  and  (2)  that  substantial  pieces 
of  management  control  documents  were  never  prepared  for  DSAMS,  and  thus  need  to  be 
prepared 

Our  overall  le^nsc  to  each  is;  (l>that  the  estimated  development  cost  of 
DSAMS,  based  on  preliminary  analysis  perfonned  by  the  Institute  for  Defense  Analyses 
(IDA),  in  advance  of  their  final  report  due  in  April  199S,  is  ^>proximately  SIOOM,  and 
(2)  that  appropriate  managemeiit  controls  were  in  place  althouj^  some  supporting 
documentation  may  not  have  followed  conventional  format  guidelinm.  Attachment  1 
contains  our  detailed  responses  to  the  ten  specific  recommendationi  in  the  draft  report 
This  is  not  to  suggest  that  the  DSAMS  project  was  withal  probtems,  but  despite  early 
missteps  that  many  systems  development  projects  experience,  the  first  phase  of  DSAMS 
has  been  successfully  implemented  in  production  and  is  already  meeting  the  needs  of  our 
user  community.  In  fact,  by  the  end  of  calendar  1998,  we  estimate  we  can  begin  to  shut 
down  two  and  most  of  another  of  the  13  legacy  systems  DSAMS  w-is  designed  to  rqilace. 

The  DSAMS  effort  is  not  without  complexity  and  risk,  but  the  Security 
Assistance  community  has  collectively  agreed  that  this  is  the  best  road  for  the  Imig  tenn. 
As  our  resources  are  directly  dependent  on  the  level  of  Foreign  Military  Sales  (FMS),  and 
as  those  sales  are  projected  to  decline  in  the  outyears,  the  investineiu  made  now  in  a 
modem,  standardized  tool  for  all  the  Military  Departments  (MILDEPs)  and  other  Defense 
Agencies  will  help  soften  the  impact  wifoout  diminishing  customer  axjpporx. 

Before  the  DSAMS  effort  started,  DSAA  provided  funding  of  r^proximately 
$40M  a  year  to  run  the  legacy  systems.  By  taking  strategic  cost  reduction  actions  such  as 
collocating  the  largest  legacy  systems  into  a  single  Defenac  Megacemer,  we  have  so  far 
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Attachments 
2  through  6 
not  included 


been  able  to  keep  tiie  legiuy  tystems  opcraCtontl  and  (o  fund  DSAMS  develt^jmeot 
^thin  the  S40M  window.  Impoitant  to  note  ts  that  if  there  were  no  DSAMS  effort,  the 
MILDEP  legacy  systems  would  need  to  be  independently  itsystemized.  The  coUective 
costs  of  those  independent  effoHs  would  hr  exceed  a  centralized  effoit  like  DSAMS  and 
the  Security  Assistance  community  would  still  be  faced  with  many  of  the  current  system 
problems  (e.g.,  lack  of  standardization,  frequent  reconciliation,  redundant  data,  and 
customer  frustration). 

In  a  memorandum  dated  November  19, 1996,  ASp(C3I)  deleted  DSAMS  from 
the  list  of  nuyor  automated  information  systems  because  its  development  cosU  were 
projected  to  be  below  $120M.  Prdiminaiy  analysis  by  IDA  suj^xts  that  cost  projection, 
and  we  expect  the  final  icsults  in  April  1998. 

As  the  attachments  indicate,  DSAA  believes  adequate  management  controls  exist 
for  providing  reasonable  assurance  that  DSAMS  is  operating  as  intended.  The  successful 
implementation  of  the  first  module  has  served  as  a  proof  of  concept,  and  we  are  confident 
the  experience  gained  will  ensure  DSAMS  proceeds  on  a  sound  footing. 

aOtoMMcKa%)  ^ 
Acting 


Copy  to: 
ASD(C31) 


Attachments: 

1  -  DSAA  Reposes  to  DoD  IG  Reconunendations  for  Corrective  Actions 

2  -  Mission  Needs  Statement  (MNS) 

3  -  Operational  Requirements  Document  (ORD) 

4  -  Acquisition  Program  Baseline 

5  -  Acquisition  Strategy 

6  -  Acquisition  Plan 
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DefeiM  Security  Aieiittuce  Agency  (DSAA)  Responses  to  DoD  Inspector 
Genertl^s  Draft  Andit  7L&0034  Recommendations 
for  Corrective  Action 


Prior  to  ■ddresring  etch  spccifkfcconuncpditkm  fa  cotrecdvcscthm,  key 
infonnatioii  sod  amlyis  nowbe  piesentel 


Cc$t 

•  Augaificaznmeccuncyrepestedmstletstfivepamofdtoiuditiqtonisth^ 

origlDal  estimate  of  S58.3M  was  the  l(fe<yeli  cost  for  ihtjlni  moduli  otify.  In  ikt,  the 
original  estimate  of  S58.3M  was  dto  esdoiated  cost  fof  mU  modules. 

Acost  table  in  the  Conceptual  Design  Document  with  '*life  cycle**  in  the  title  but  with 
details  related  to  the  phasn  of  the  software  development  Ufe<ycle  may  have  confused  the 
auditors.  Recommend  making  those  coriections  in  the  final  report 

•  The  estimated  S500M  life-cycle  cost  for  DSAM5  is  significantly  overstated  Theitpoti 
states  that  ASD(PAAE)  used  the  SEER  software  estimation  mo^.  The  IC  infonned 
DSAA  that  die  only  inputs  to  this  model  were:  (I )  lines  of  code  (provided  by  DSAA)»  and 
(2)  a  10-year  life  cycle;  otherwise,  all  defauh  parameten  were  assumed. 

■  The  life-cycle  calculations  were  based  on  a  2.5M  lines  of  code  size  estimate  followed  by  a 
10-year  life-cycle.  These  inpuu  will  lead  to  a  S500M  estimate  if  one  assumes  a  nominri 
SlOO  per  line  of  code  development  cost  (S250M  for  development)  and  a  oominal  IO*/» 
operations  and  support  (OAS,  or  maintenance)  cost  per  year  for  10  years  ($250  for  OAS). 

•  This  analysis  is  inaccurate  for  several  reasons.  First,  the  lines  of  code  were  inflated.  We 
provided  the  2.5M  figure,  but  that  estimate  was  based  solely  on  legacy  systems,  and  we 
did  not  allow  credit  for  reus^le  application  code,  non-recurring  reference  and 
infrastructure  code,  automatic  code  generation,  higher  level  languages,  and  code  being 
eliminated  by  DFAS. 

Although  it  is  difficult  to  measure  all  the  legacy  systems  in  a  consistent,  meaningful 
way,  a  more  accurate  estimate  can  be  deteimined  by  adding  together  all  the  systems  of 
a  single  MILDEP  that  cover  the  full  nmgeofFMS  functionality  in  DSAMS.  For 
example,  if  fines  of  legacy  code  for  ail  the  Navy  FMS  systems  (NSADS,  MISIL, 
STATIS)  arc  added  together,  then  adjusted  (up  and  down)  for  DFAS  functionality,  and 
then  inflated  by  lOOK  to  cover  unknowns,  the  answer  is  approximately  liM  lines  of 
code.  Fimher,  based  on  actual  experience  with  the  Case  Dmlopment  Module, 
DSAMS  contains  approximately  1  fine  of  code  for  every  1.5  lines  of  legacy  code, 
vriuch  would  result  in  the  proje^  total  fines  of  code  in  DSAMS  as  IM  vice  2.5M. 

•  Second,  a  more  accurate  costing  can  be  done  using  lines  ofcode  developed  and  actual 
cosu  expended  so  far  and  extrapolating  to  subsequent  modules.  The  fiuti  report  from 
IDA  will  incorporate  tfab  analyss,  but  tite  peefiminary  analysis  so  far  indicates  a 
development  cost  of  approxunttely  SIOOM. 
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•  THid,  ifthecoitaiijdysUmtfwdi«ftrepoiluaccepted,itcou3db«crossdieckedfor 
rough  iccuncy  against  whit  has  actually  been  developed  ukI  spent  to  date.  The 
preliminaiy  aiUlysis  shows  considerable  variance,  and  a  fum  comparison  will  be  possible 
after  the  rmits  of  die  final  IDA  report  are  Ttccived. 

•  Several  events  in  eariy  FYI 997  caused  DSAA  itself  to  question  the  projected  cost  of 
DSAMS.  For  example,  the  functiocud  transfer  of  MELDEP  personnel  to  staff  the  Defense 
Security  Assistance  Development  Center  (DSADC)  occurred  one  year  later  than  planned, 
and  it  became  obvious  that  costs  would  be  higher  than  originally  projected.  Theseevents 
led  to  the  dedsioa  to  retain  IDA  for  an  independent  assessment  of  projected  costs. 


ScAcifn/e 

•  The  report  incorrectly  claiins  that  **DS  AA  fielded  four  versions  of  the  case  development 
module  that  were  not  fully  operational,  resulting  in  program  slippages  from  May  1999  to 
beyond  the  year  2000**.  Only  one  version  oftbeCDM  has  been  fielded  Into  prx^uction. 
The  rqiort  is  referring  to  the  first  version  in  March  1 997  where  it  became  evident  during 
user  letting  that  additional  ftmctiooality  wu  required  in  <Rder  for  the  users  to  accomplish 
die  case  developineoi  mission  whh  the  same  ef&ieacy  as  the  legacy  systems.  Becauseof 
proper  mansgement  controls  to  ensure  software  is  not  released  into  produetkm  if  it  does 
not  meet  user  needs,  DSAA  re-evaluated  the  schedule  accordingly.  At  this  point  it  was 
clear  that  the  original  1999  DSAMS  completion  date  was  too  optimistic. 

After  the  ^^lication  was  subsequently  modified  and  tested  by  users  in  September 
1 997,  both  die  Navy  and  the  Army  sg^  to  go  with  that  version  into  initial 
opentionil  capability  0OC).  Primarily  because  of  problems  converting  legacy  data, 
ibc  CDM  IOC  release  was  re-scbeduled  but  is  now  in  production.  Important  to  note  is 
that  during  diis  period,  the  application  itself  was  satisfiKtoiy  to  the  users,  and  the  time 
was  used  to  iiKorporaie  enhancements  identified  as  important  by  the  users  and  to 
make  perfcmnance  enhancements  to  the  application  and  communications. 

■  TIk  Estimated  Fielding  Date  table  in  the  report  shovrs  4  dates  based  on  4  difTcient  source 
documents.  The  third  date  shown,  30  Nov  2004,  is  much  higher  than  the  others  and  gives 
the  appearance  of  wild  swings  in  the  DSAMS  schedule.  In  fact,  the  source  document 
cited  shows  the  fielding  of  the  last  module  to  be  estimated  for  31  Aig  1999,  consistent 
with  the  previous  estimate;  the  2004  date  was  the  projected  end  of  the  system  life  cycle. 
The  auditors  nuy  have  bea  confused  by  the  source  docurnent  Recommend  making  this 
change  to  the  tiMe  in  the  final  report 

•  The  claim  dm  DSAMS  slippages  resulted  in  additional  legacy  systems  costs  due  to  Year 
2000  compliance  is  highly  misleading.  Completing  DSAMS  prior  to  the  Year  2000  to 
avoid  ^blems  with  legacy  systems  not  being  compliant  was  never  a  fundamental 
objective  of  DSAMS;  rather,  it  was  aiother  potential  benefit.  If  there  wu  no  DSAMS, 
Year  2000  costs  could  not  be  avoided,  so  their  discussion  in  this  report  serves  no  purpose. 
All  Security  Assistance  legacy  systems  are  foHowing  die  DoD  Y2K  Management  Plan. 
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•  In  eariy  FY1997  DSAA  rctlized  that  die  original  schedule  of  1999  was  too  optimistic. 
Since  that  time,  September  2001  has  been  used  as  the  estimated  dale  for  the  final  module 
to  be  initially  deployed.  The  schedule  will  be  re-evaluated  after  the  IDA  report. 


TcsriNg 

•  The  stateroem  that ‘"diving  Air  Fofte  testing  of  the  case  devek)pinemin^^ 
om  300  deficlettties  were  identified""  is  m^leading.  Oftbe  total  number  of  problems 
itported  by  the  Air  Force  during  testing  of  the  apphcatiofi,  approximately  25H  eventually 
resulted  in  no  action,  stnce  they  were  due  to  such  problems  as  inoorrect  use  of  the  system, 
di^licates,  or  lenqxniycoinmuriicatioiu  or  coofiguratioD  issuer  ^jproximatdy  half  the 
problem  reports  were  legitimate  bugs,  all  ofwfaidi  were  subsequently  ooirected  Thefinal 
25H  of  the  problem  reports  were  requests  for  changes  to  the  d^gn  of  the  software  based 
on  new  user  requirements  not  previously  identified  Many  enhancements  identified  by  the 
users  as  impoitanl  have  since  been  Incorporated.  The  rest  have  been  documented  and  will 
be  incoiporUed  in  subsequem  releases. 


Btnifia 

Considerition  should  be  given  to  the  benefits  ofDSAMS  which  will  also  reduce  costs.  Three 

Kcem  examines; 

•  Chamci^  one  systtm  insttad  of  mMt^t  oms.  When  changes  to  policies  and  procedures 
ait  made,  those  changes  only  need  to  be  made  to  one  system.  For  example,  in  December 
1997  some  of  the  legri  woit^  in  the  Letter  of  Offer  ^  Acceptance  (the  primary 
document  between  the  U.S.  Government  and  a  foreign  govemmern  for  providing  material 
and  services)  changed.  DSAMS  produces  this  document,  and  the  diange  was  quickly 
made  in  a  single  system  nfoer  than  in  each  MILDEP  system.  A  further  advanage  is  that 
the  potential  for  mconslMfU  changes  among  multiple  systems  is  eliminated. 

•  Eiiminaiing  outdared  business  processes.  Gathering  the  MILDEPs,  DLA,  DF  AS  and 
DSAA  together  to  attempt  to  standardize  and  ie<engineer  security  assisunce  business 
practices  is  fundamental  to  meeting  the  challenges  ofoutyetr  reductions  in  the  FMS 
Administrative  Trurt  Fund.  At  these  fenmu  outdated  policies  and  procedures  are 
challenged  and  sometiznes  changed  or  diminated.  A  recent  example  mvoives  Cooperative 
Logistics  Si^y  Support  Airangements  (CLSSAsX  a  Qfpe  of  FMS  case  that  each 
MILDEP  executes  t^dificfcntly.  In  negotiating  ways  to  standardize,  a 
itcommendadon  fim  the  group  (comprisLog  lepresentativei  of  all  MIU)EPs,  DLA, 
DFAS  and  DSAA)  waste  elimmate  CLSSAs  entirely,  as  they  have  outlived  their 
usefulness.  If  approvd  to  diminate  is  granted,  the  scope  ofDSAMS  will  be  reduced 
accordingly. 

•  Btisimss  process  n-tngineering,  TlieNavy  is  already  taking  advantage  of  the  availability 
ofDSAMS  to  re-engineer  the  case  development  process.  For  the  first  time  they  are  able 
to  de-centrdize  the  production  of  Letters  of  Offer  and  Acceptance  which  will  ultimately 
result  in  improved  customer  service  and  reduced  costs. 
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Revised 


Revised 


Other  C0frtethtu  Nuded  At  the  DntflAtiiit  KtpoH  (not  otherwise  discussed) 

1.  Page  X  paragraph,  2^  line  from  end:  The  prognun  management  office  was  established 
in  September  1995,  using  t  people  detailed  horn  each  MILDEP  and  DFAS  (2  each),  and 
not  in  April  1997  as  die  itpoit  ideates. 

Z  Page  13,lastpiragrq)h,lastllne:  Astn&atedtothelGonDecember  10, 1997,before 
the  draft  report  was  forwarded,  the  IDA  cost  analysis  will  not  be  available  in  January 
I99S.  IDA  expecutocomplc^  its  final  report  to  DSAA  in  April  199S. 
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In  conjunction  with  the  key  bickgrouad  infomution  ind  analysis  pitsented  earlier,  DS  AA 
responses  to  each  of  the  DoD  In^teclor  Ccneral^s  recommendations  for  corrective  action  are 
adtosed  as  follows: 


RecoMmendadoa  2  a:  Submit  the  life-cycle  cost  estimate  prepared  by  IDA  to  ASD(C3I)  for 
reclassification  of  DSAMS  as  a  major  automated  information  system. 

Koa-eonear.  This  recommendatioa  assumes  the  S500M  life-cycle  cost  is  valid. 
Although  IDA'S  final  report  is  not  due  to  DSAA  until  April  199S,  the  pretimiaary  analysis 
indicates  DSAMS  costs  are  under  the  MAISRC  threshold  and  ASD(C31)  already  agr^  in  a 
19Novcmbcr  1996  memorandum  that  DSAMS  is  not  a  MAISRC  system,  based  on  cost 
estimates  we  provided  earlier. 

The  order  ofRecommendatioQs  2a  and  2b  dxHiId  be  reversed.  Ifthe  final  report  from 
IDA  is  validated  to  be  over  the  MAISRC  threshold,  then  it  would  be  iq^prapriate  to  submit  the 
cost  estimates  to  ASD(C3I)  for  reclassification. 

Rfcommcndatlou  2  b:  Submit  the  life-cycle  cost  estimate  prepared  by  IDA  to  ASD(PA&E) 
for  validation  and  accuracy  and  completeness  of  the  life<ycle  cost  estimates. 

Coacur.  IDA*s  final  report  is  due  to  DSAA  in  April  1998. 

RfcommendatioB  2  c:  Update  the  life-cycle  cost  estimate  for  each  milestone  review  so  that 
DSAMS  can  be  evaluated  for  itt  cost-effectiveness. 

Coucur.  After  receiving  the  final  report  fiom  IDA,  DSAA  will  ensure  costs  are 
regularly  tracked  and  evaluated. 

Recommendatlou  2  d:  Prepare  a  mtiskm  needs  statement  for  DSAMS  to  identify  usets’ 
needs,  describe  the  deficiencies  in  foe  existing  systems,  and  identify  potential  alternatives. 

Complete.  The  requirements  of  a  mission  needs  statement  (MNS)  were  followed  and 
documented,  although  no  document  in  foe  conventiona]  MNS  format  was  previously 
produced.  However,  a  MNS  in  foe  required  format  is  attached. 

Rccemmcttdatioa  2  e:  Prepare  an  opcradonal  requirements  document  for  DSAMS  to 
esublish  system  cost,  schedule,  and  peiformance  parameters  based  on  the  user  requirements. 

Complete.  The  requirements  of  an  operational  requirements  document  (ORD)  were 
followed  and  documented,  although  no  document  in  the  conventional  ORD  format  was 
previously  produced.  However,  an  ORD  In  foe  required  format  is  attached. 

RccommendatioB  2  f:  Establish  an  integrated  product  team  to  provide  oversight  and  review 
as  DSAMS  proceeds  through  iu  acquisition  life  cycle.  The  integrated  product  team  should 
include,  at  a  minimum,  rQvescnualves  from  ASD(C3IX  OFAS,  ASD(PAd^£).  and  the 
MILDEPs. 
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See  chart  on 
page  37 


CMpkt*.  Ahbough  w  11019  ii  ibfnally  libeled  in  "IFr,  tte  nquitcawiitt  of  M 
ittegnied  iModoct  teiin  hm  been  feUowed  duoiviKM  the  developinem  of  DSAMS. 
DSAMS  his  eaj^  ud  depended  on  the  AiU  pertidpetioa  of  dw  MILDEPs,  DFAS  end 
DLA  in  numerout  teems.  Ibe  Executive  Steering  Committee  saves  as  the  oventching 
integrated  product  team,  which  it  a  nuhidiie^M  teem  working  Uravd  1  common  goal 
with  both  the  lespemibiliiy  and  authority  for  ptogram  dcciiiont.  Hk  ESC  is  eempoied  of 
representatives  fiom  ail  appropriate  diseipliaet  working  together  to  build  successful  and 
balanced  programs,  identify  and  resolve  itaues,  and  make  sound  business  dedsioos. 

The  ESC  is  «9pocted  by  a  VHiety  of  other  teams,  including  Conceptual  Design, 
General  Design,  and  StandardizMion  Working  groups  charged  whh  re-engineering  the 
security  assistance  business  proetsset,  such  at  ease  development,  case  inqrlementetion, 
training  and  case  execution.  A$D(C3I)  was  never  included  in  the  past  because  milestone 
decision  authority  was  defeated  to  DSAA.  The  overall  DSAMS  organizatiott  structure  is 
shown  below. 


Developing  an  automated  system  that  satisfies  the  security  assistance  businets 
requirements  fbr  all  three  MILDEPs  is  ehalltngin^  Aldwugh  the  Security  Assistance 
Management  Manual  (SAMM)  gives  guidelines  for  meeting  the  legal  tequitements  ofUS 
foreign  tnilitaiy  assistance,  the  MILDEPs  all  develaped  btitpuAnt  processes  and  systems 
for  developing,  implementing  and  executing  securi^  assistance  cases.  In  order  to  ensure  that 
the  different  business  processes  now  in  operation  within  the  MILDEPs  did  not  drive  dK 
DSAMS  software  development  effort  to  building  redundant  aqribilities  within  the  software, 
the  PMO  established  the  following  process  fbr  building  the  system: 

•  Business  Process  Standardiatioo.  MILDEP/DFAS/DLA  representatives 
empowered  to  make  changes  in  dieir  current  busineu  processes  and  procedures 
meet  to  leview.  re-engineer  and  standardize  the  US  Security  Asastance  process. 

BwairaoMwtDDOoataiaorTioooM  a 
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Defense  Security  Assistance  Agency  Comments 


Final  Report 
Reference 


•  ReqittrementsDefinitioo 

•  Software  Design/Ciieclq)oiiit$ 

•  User  Testing 


Attachment 

deleted 


Attachment 

deleted 


Attachment 

deleted 


RecomacidatloB  2  g:  Establish  a  program  baseline  for  DSAMS  to  manage  and  control  cost 
growth  and  schedule  slippages. 

Coaeitr.  Because  the  successful  implementation  of  the  Case  E)evelopment  Module  is 
serving  as  a  proof  of  concept,  a  baseline  for  performamre  ami  schedule  is  being  establish^  it 
ihatpoint  Cost  will  be  added  to  the  basdinc  after  the  final  IDA  report  is  submitted  to  DSAA. 
The  program  baseime  is  attached. 

RecMmeadatiM  2  h:  Prepare  an  overall  acquisitipQ  strategy  for  DSAMS  lo  mani^  user 
requirements  atd  klentiiy  the  system’s  Ufe  cycle  and  the  resources  to  be  used. 

CampiKc.  The  lequiremeritsofaD  acquisition  strategy  were  followed  and 
docui&ented,ahboqgh  no  docunieiU  actually  laMed  as  such  was  previously  produced. 
However,  an  Acquisition  Strategy  in  the  required  format  is  attached. 

Rmmmendatfon  Z I:  Prepare  an  acquisition  plan  for  DSAMS  to  tdendfy  the  risk  associated 
with  cost,  schedule,  and  perTonnatiee  parameters  u  well  as  solutions  to  those  risks  so  that 
prognm  goals  can  he  met. 

Complete.  The  requiiements  of  an  acquisition  plan  were  followed  and  documented, 
although  no  document  actually  labeled  as  such  was  previously  produced.  However,  an 
Acquisition  Plan  in  the  required  format  is  attached. 

Rmmmendation  2  j:  PiqNue  a  test  and  evaluation  master  plan  for  DSAMS  to  establish  the 
criteria  for  testing  the  system  to  verify  that  user  requirements  are  met. 

Snpemded.  A  Software  Test  Descriptiem  and  a  Software  Test  Plan  were  produced 
for  the  Case  Development  Module.  These  documents  followed  the  standards  of  MIL  STD 
498  and  me  at  a  more  detailed  level  than  a  conventional  test  and  evaluation  master  plan.  We 
will,  however,  prepare  a  test  and  evaluation  master  plan  (TEMP)  for  each  future  module  of 
DSAMS.  TEMP  Target  Dates; 

Caseimplemeitation:  30Sep98  Case EiecvtioB:  30Sep99 

Training:  30Nov98  CaseClosnre:  30Scp99 


DSAARmppims  to  DOO  IQ  nvert  71/00094  7 
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Department  of  the  Air  Force  Comments 


DEPARTMENT  OP  THE  AIR  FORCE 

WAtlMWOTOWtOC 


29  Jinuuy  1998 


MEMORANDUM  FOR  DODIG  (ATTN.  MARY  GEIGER) 

FROM:  SAF/FMBIS 

1 130  Air  Force  Pentagon 
Washington,  DC  20330-1130 

SUBJECT:  Draft  Audit  Report  on  the  Defense  Security  AssisUmcc  Management  System 
(Project  No.  7LG-0034) 

We  have  reviewed  the  draft  audit,  and  concur  as  follows: 


RecommeDdation  I  -  Concur 
Recommendation  2  • 


a«,  b.,  c.  -  Concur 

iL  «  Concur  •  The  mission  needs  statement,  intended  to  ..identify  users 
needs...**,  should  include  requirements  to  serve  both  the  Financial  Maiuigement  and 
Accoundng  communities. 

c.  •  Concur  -  Financial  Management  and  Accounting  requirements  should  also  be 
established  as  perfommee  parameters  in  the  Operational  Requirements  Document. 

t  -  Concur  -  It  is  critical  to  have  DFAS,  and  selected  members  of  the  Financial 
Management  community  on  the  IPT,  to  ensure  the  items  above  are  addressed. 


g.,  h.,t,J.- Concur 


SAF/FMBIS  points  of  contact  are 
(695-5979). 


Robert  Rosenbaum  (614-5340),  and  Sharon  Tucker 


f  danaJ^gilmour 


AssisUmt  for  Security  Assistance 
Deputy  Assistant  Secretary,  (Budget) 
Assistant  Secretary  of  the  Air  Force 
Finmcial  Management  A  Comptroller 
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Audit  Team  Members 


The  Readiness  and  Logistics  Support  Directorate,  Office  of  the  Assistant 
Inspector  General  for  Auditing,  DoD,  produced  this  report. 

Shelton  R.  Young 
Raymond  D.  Kidd 
Evelyn  R.  Klemstine 
Priscilla  G.  Sampson 
James  B.  Mitchell 
Bryon  J.  Farber 
Garry  Durfey 


INTERNET  DOCUMENT  INFORMATION  FORM 


A .  Report  Title:  Defense  Security  Assistance  Management  System 

B.  DATE  Report  Downloaded  From  the  Internet:  09/23/99 


C.  Report's  Point  of  Contact:  (Name,  Organization,  Address,  Office 
Symbol,  &  Ph  #):  OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General,  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-2884 


D.  Currently  Applicable  Classification  Level:  Unclassified 


E.  Distribution  Statement  A:  Approved  for  Public  Release 


F.  The  foregoing  information  was  compiled  and  provided  by: 
DTIC-OCA,  Initials: _ VM _ Preparation  Date  09/23/99 


The  foregoing  information  should  exactly  correspond  to  the  Title,  Report  Number,  and  the  Date 
the  accompanying  report  document.  If  there  are  mismatches,  or  other  questions,  contact  the 
above  OCA  Representative  for  resolution. 
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